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We study the phase diagram of two-dimensional Bose-Fermi mixtures of ultracold atoms on a 
triangular optical lattice, in the limit when the velocity of bosonic condensate fluctuations is much 
larger than the Fermi velocity. We contrast this work with our previous results for a square lattice 
system in Phys. Rev. Lett. 97, 030601 (2006). Using functional renormalization group techniques 
we show that the phase diagrams for a triangular lattice contain exotic superconducting phases. 
For spin-1/2 fermions on an isotropic lattice we find a competition of s-, p-, extended d-, and /- 
wave symmetry, as well as antiferromagnetic order. For an anisotropic lattice, we further find an 
extended p-wave phase. A Bose-Fermi mixture with spinless fermions on an isotropic lattice shows 
a competition between p- and /-wave symmetry. These phases can be traced back to the geometric 
shapes of the Fermi surfaces in various regimes, as well as the intrinsic frustration of a triangular 
lattice. 



PACS numbers: 03.75.Hh,03.75.Mn,05.10.Cc 



I. INTRODUCTION 

During the last decade there has been remarkable 
progress in controUing and manipulating ensembles of 
ultracold atomsii^i^. Using optical lattices and dif- 
ferent types of atoms, a variety of non-trivial many- 
body states can be 'engineered' that are very accu- 
rate realizations of Hubbard models. The experimen- 
tal observation of the Mott-Hubbard superfluid-insulator 
transition^, of fermionic superfluids^, of the Tonks- 
Girardeau gas^ and other Luttingcr liquids^, the mea- 
surement of noise correlations'^, instabilities^, as well 
as localization effects^" in Bose-Fermi mixtures (BFM), 
have created a great deal of excitement in the physics 
community. These findings have prompted a large 
amount of theoretical work that study charge density 
wave (CDW) order—, many-body states of composite 
particlesi^, polaronic effectsi^, and tuning of Feshbach 
resonances on latticeai^. 

It is known that the geometric shape of the lattice is 
a crucial factor in determining the properties of inter- 
acting many-body systems. For instance, localized spins 
interacting antiferromagnetically on a triangular lattice 
suffer from the phenomenon of frustration, when anti- 
ferromagnetic order cannot be achieved because of the 
particular lattice structure. For itinerant fermionic sys- 
tems, the lattice structure, together with the dispersion 
relation and the filling fraction, determine the shape of 
the Fermi surface (FS). The FS, by its turn, is a crucial 
factor in determining what type of orders the system can 
develop. Indeed, for the triangular lattice we consider in 
this paper, which shows a rich and subtle competition 
between superconducting phases with different symme- 
tries, we find that small changes in the shape of the FS 
determine which pairing symmetry is dominant. This is 
a reflection of the "lattice frustration" on the supercon- 
ducting phases. In solids, this intriguing lattice geometry 



is realized in materials such as cobaltatesii, transition 
metal dichalcogenides^ and k-(ET)2X layered organic 
crystalsi2, (if each lattice site is represented by one ET 
dimer— ), and has been the subject of several theoretical 
studies21ii22i. 

The versatile technology of optical lattices allows to 
create and tune a vast number of lattice geometries for 
ultracold atoms, and is therefore a promising experimen- 
tal environment to study the rich many-body physics that 
can be engineered within these systems. We study mix- 
tures of ultracold fermionic atoms (either with two hy- 
perfine states that are labeled by pseudo-spin "f and |, or 
single component, spinless, fermions), and single compo- 
nent bosons on a triangular lattice. Such a system can be 
realized as a ^°K-^^Rb mixture in an optical lattice cre- 
ated by Nd:YAG lasers, with tunable interactions. The 
geometry of the lattice under consideration is shown in 
Fig. Ha). 

This paper is organized as follows: In section |TT] we de- 
rive the effective Hamiltonian for the fermionic degrees 
of freedom, in the limit of fast condensate modes. In sec- 
tion |TTT] we describe the functional renormalization group 
approach and its implementation. In section IIVI and |V] 
we describe BFMs with spin 1/2 on an isotropic and an 
anisotropic lattice, respectively, and in section lVll we con- 
sider spinless fermions. In the last section IVIII we com- 
ment on the detection of the phases found in this paper, 
and conclude. 



II. EFFECTIVE HAMILTONIAN 

BFMs in optical lattices are well described by a Hub- 
bard model of the form. 
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i i 

+ ^nb^inb^i + UbfUb^inj^i'^ , (1) 

where //^ {fi,s) creates (annihilates) a fermion at site i 
with pseudo-spin s {s bl (bi) creates (annihilates) 

a boson at site i, nj^i = flsAs {nb,i = h\hi) is the 
fermion (boson) number operator. With this Hamilto- 
nian we describe a BFM with spin 1/2 fermions, whereas 
a Hamiltonian of a BFM with spinless fermions can be 
obtained from this Hamiltonian by suppressing the spin 
indices. Here we allowed for two different values for the 
hopping amplitudes, for the two types of lattice bonds 
and (ij)2- For a triangular lattice, tf^a and tb,a with 
a = 1,2 are the fermionic and bosonic tunneling ampli- 
tudes between neighboring sites, where the index a = 1 
[a — 2) refers to the continuous (dashed) bonds, as shown 
in Fig. [2] a). For the description of the isotropic case 
we equate tb//,i and tb/f,2i and define tf = tf^i = tf^2 
and tb = tb^i — tb,2- l^f (p-b) is the chemical potential 
for fermions (bosons), Ubb, Uff, and Ubf are the on-site 
boson-boson, fermion-fermion and boson-fermion repul- 
sion energy, respectively. 

We consider the limit of weakly interacting bosons, 
in which the bosons form a BEC, for which we use the 
standard Bogoliubov description. Specifically, we assume 
that a macroscopic number Nq of bosons is condensed in 
the fc = mode, which we take into account by replacing 
the operator bo by \/No. We then keep all the remain- 
ing terms up to second order in the operators b^, with 
k 7^ 0, which includes anomalous terms corresponding to 
pair creation and annihilation from the condensate, and 
diagonalize this expression via a Bogoliubov transforma- 
tion. The resulting dispersion relation is given by 



= y (eb,k - eb,o)(eb,k - £6,0 + 2J76(,n6), (2) 
where the bare lattice dispersion is given by: 

Cb.fc — — i6.l2 COS fca, — tb^2(2 COs(fca;/2 + \/3fcj,/2) 

+2 cos{k^j2 - Viky/i)). (3) 
For small values of and ky, Wk can be expanded as: 

t^k - \l {{2tb,i + tb,2)kl + 3tb,2kl)Ubbnb, (4) 
which gives us the two velocities 



Vb^x = y i2tb,i + tb^2)Ubbnb (5) 
Vb,y = \/3tb,2Ubbnb. (6) 

In addition to the assumption that the bosons form a 
BEC, we assume that these velocities of the conden- 
sate fluctuations are much larger than the Fermi veloc- 
ity, which corresponds to the conditions Vb^x/y > ^f,i/2- 
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FIG. 1; a) Fermi surfaces for an isotropic lattice, for < 
2i/,i/2) M/ — (hexagonal shape), and yU/ > 2ty 1/2 (six 

disjoint arcs) . b) Diagram of the different types of Fermi sur- 
faces that can be created on an anisotropic lattice, by varying 
the ratio t/,2/t/,i and ^f. c) Fermi surfaces for t/,2 < for 
M/ < 4t/,2 - 2f/,i, 4t/,2 - 2t/,i < A*/ < 2f/,i, and ^i/ > 2f/,i, 
corresponding to the regimes I-III, respectively, d) Fermi sur- 
faces for t/,2 > for /i/ < 2t/,i, 2t/,i < /x/ < 4f/,2 — 2f/,i, 
and /i/ > 4t/_2 — 2t/_i, corresponding to the regimes IV- VI, 
respectively. 



Therefore, large bosonic hopping amplitudes, a bosonic 
density of « 1-3, and some intermediate value for i/,i/2 
will satisfy this requirement. In this limit the bosonic 
modes can be integrated out, and one obtains an approx- 
imately non-retarded fermion-fermion interaction. The 
induced potential I4nd.,k is given by: 

T4nd.,k = -l^/(l-K^?(2-2cosfc,) 

+e|(4 - 4cos(fc,/2) cos(V3fcy/2))) , (7) 

with V = Ubf /Ubb, and are the healing lengths of the 
Bose-Einstein condensate (BEC) and are given by — 
\/tb_a/'^nbUbb with a = 1, 2. In summary, we arrive at a 
purely fermionic, non-retarded description of the form: 

i?off. = I] U^k - M/) fLsf^^- + ^P/,k,TP/,-k,i 

k I s 

+ ^V'ind.,k P/,k,P/ -k| , (8) 

This is the effective Hamiltonian that we study with a 
numerical implementation of the functional renormaliza- 
tion group. 



III. FUNCTIONAL RENORMALIZATION 
GROUP 

In this section we describe the numerical implemen- 
tation of functional RG^!^ that we use to study the 
effective Hamiltonian ([8]). In this approach we integrate 
the one-loop RG equations for the four-point interaction 
vertex [/(ki,k2,k3), which arc functions of the direction 
of the momenta on the FS, with the fourth momentum 
not explicitly written, given by = ki + k2 — ka . We 
solve the RG equations for all the C/(ki, k2, ka)^, given 
by: 

9^;7^(ki,k2,k3) = 

- / \di[Gi{p,uj)Gi{qi,u)]Ui, (p, ki, k4)C/^~, (p,k3,k2) 

+5^[G^(p, c^)G,(k, -t^)]{/,-Jki, k2, p){/,-Jk3, k4, p) 

+dt[Gt{p, c^)G,(q2, u;)] {-2(7,~^^(ki, p, \^^)Ui^j^U, p, k2) 

+ Ul^^,'kx ,k3) Ui^^4. ,P,k2) + ?7^~^ jki ,p,k3) C/^~^^^,k4,k2)}}, 

(9) 

where £ = ln(Ao/A) (Aq ~ 6^/ is a high-energy cut-off), 
k = ki k2 - p, qi = p -h ki - k4, 92 = p ki - ks, 
Ipp = min{£p,4}, = m\n{ip,tq^},iph = m:u\{£p,iq^}, 
Ip^ = ln(Ao/|ep|), and G,(k,u;) = 6(1^1 - A)/{iu; - ^k) 
with = £f,k — fJ-f- The initial, i.e., bare value, of 
C/(ki,k2,k3) is given by: 

C/(ki,k2,k3) = f///+^nd.,ki-k3- (10) 

From the vertex [/(ki,k2,k3) we obtain the possible 
many-body instability channels such as CDWi, spin den- 
sity wave (AFi), where the index i refers to the three 
nesting vectors, and the superconducting (BCS) channel. 
These channels are given by: 

yCDW^ ^ 4C/,(ki,k2,ki + Q0, (11) 

V^^' = 4C/,(ki,k2,ki+Qi), (12) 
T/^^^ = {/(ki,-ki,k2), (13) 

For the isotropic case, perfect nesting occurs at 3/4- 
filling, with three possible nesting vectors: Qi = (0, 27r), 
Q2 = (tt, v^tt), and Q3 — (— tt, VStt), leading to three 
different possible types of instabilities per density wave 
channel. For the anisotropic case, only Qi can be a nest- 
ing vector, for the condition fif — 21 f i. 

To determine the scale of the gaps. A, associated with 
each of these order parameters, we use a 'poor man's' 
scaling estimate, specifically: A w Aoe~^'=, where £c is the 
point at which the RG flow diverges and the instability 
occurs. 

The RG is implemented numerically by discretizing the 
FS into M = 24 or 36 patches. The CDW and AF chan- 
nel is represented by an (M/3) x (M/3) matrix, the BCS 
channel is represented by an M x M matrix, which are 
diagonalized at each RG step. The dominant instability 
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FIG. 2: a) Lattice geometry of the system. The continu- 
ous (dashed) bonds correspond to the hopping amplitudes 
tb/f. 1(2)- For tfc//,! = tb/f,2, the lattice is an isotropic tri- 
angular lattice, b) schematic representation of the AF order 
corresponding to nesting vector Qi. c) -I- d) Order parameters 
of the extended d-wave orders Di and D2 . e) Order param- 
eter of the /-wave phase. This order can also be interpreted 
as two s-wave paired hole states whose order parameters are 
out of phase by tt. f ) Order parameter of the extended p-wave 
phase, that appears in anisotropic lattices. 

is the channel that has an eigenvalue (divided by the di- 
mension of the matrix) with the largest magnitude (for 
BCS one has to ensure that such eigenvalue is negative so 
that the channel is attractive). Each element of the cor- 
responding eigenvector represents a given FS patch, and 
hence, the symmetry of the dominant order parameter is 
reflected on the patch (i.e., angular) dependence of each 
element around the FS. Using this method, we determine 
the phase diagram of the system in various limits. 



IV. ISOTROPIC LATTICE 

We first consider spin- 1/2 fermions on an isotropic tri- 
angular lattice, i.e. with tf^i = tf^2 = tf- The FS for 
such a lattice behaves as follows: For small filling the 
FS consists of one near-circular piece, which then ap- 
proaches the shape of a hexagon as ^/ approaches the 
special value fXf = 2tf. At this special chemical poten- 
tial, which corresponds to 3/4-filling, the FS is nested 
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with the three distinct nesting vectors Qi. For filhng 
fractions larger than 3/4 the FS breaks into six disjoined 
arcs. Examples for these different regimes are shown in 
Fig. [T]a). Without coupling to the BEC, the fermions 
form an s-wave pairing phase for attractive interactions, 
and a Fermi liquid phase for repulsive interactions^^, ex- 
cept for the specific case fj,f = 2tf, where the system 
shows AF order for repulsive interactions. A schematic 
picture of this order is shown in Fig. [2] b) for the nesting 
vector Qi. 

We found a similar behavior for an isotropic square lat- 
tice in 16], s-wave pairing for attractive interaction, and 
Fermi liquid behavior for repulsive interaction, except at 
half-filling, for which we find AF order. An interesting 
diff'erence for the triangular lattice is the three-fold de- 
generacy of the AF phase, an indication of frustration. 

When one turns on the coupling to the BEC, the 
isotropic system shows a phase diagram of the type shown 
in Fig. [21 The s-wave pairing phase slightly extends 
into the regime of positive [///, because of the induced 
attractive interaction mediated by the bosonic fluctua- 
tions. The regime that showed Fermi liquid behavior in 
the absence of the induced interaction now shows a rich 
competition of various types of pairing. In the regime 
where the density is below half-filling, when the FS is 
approximately circular, the system shows p-wave pair- 
ing. For fillings larger than 3/4, when the FS consists of 
six disjoined parts, the fermions Cooper pair in a super- 
conducting state with / symmetry. As shown in Fig. [2] 
e), the FS in this regime can also be interpreted as two 
distinct near-circular FSs of holes. In this interpretation 
each of these two fermionic systems is in an s-wave pair- 
ing phase, but the relative phase between the two order 
parameters is tt. At 3/4- filling and large values of U/f, 
the system still shows AF order. However, for smaller val- 
ues of Uff, and also for smaller fillings, two phases with 
degenerate extended d symmetry develop. These super- 
conducting orders have a sizeable g-wave component and 
are approximately given by: 



ipm = sin26l + 0.5sin46l 
ipD2 = cos 26* - 0.5 cos 46* 



(14) 
(15) 



These order parameters are shown in Fig. [2] c) and d). 
The shapes of the order parameters are energetically ad- 
vantageous because, on the one hand, the order parame- 
ter maxima are located at points at which the system has 
a high density of states (the 'corners' of the FS). Hence, 
when the superconducting gap opens, there is a large gain 
of condensation energy coming from these regions on the 
FS. On the other hand, the d-wave state has lower kinetic 
energy than the /-wave, and hence is selected. 

The phase diagram Fig. [3] has a number of similarities 
to the phase diagram for a BFM on a square lattice, such 
as the s- and the p-wave pairing phase, and the existence 
of AF order for a nested Fermi surface for large Uff. 
However, the competition of pairing orders for positive 
Uff and intermediate and large filling is much richer, due 
to the more complex shape of the Fermi surface. 



p-BCS 




d-BCS 



s-AF 
f-BCS 



s-BCS 



3/4 



n 



FIG. 3: Phase diagram of a Bose-Fcrmi mixture on a 2D 
isotropic triangular lattice. The vertical axis corresponds to 
the interaction strength, Uff/tf, whereas the horizontal axis 
corresponds to the filling fraction of the fermions per site, n. 
The other parameters are given by V/tf = 3, and = 1 with 
a = 1,2. 



The energy gaps associated with these order param- 
eters can be determined as we did in p^ . by using a 
'poor man's' scaling argument. We find for the s-wave 
pairing and the AF order, that they are around O.ITf, 
where Tp is the Fermi temperature of the system. For 
most of the exotic phases, we energy gaps of the order of 
0.01 - O.OOlTp. 



V. ANISOTROPIC LATTICE 

We now consider a BFM with spin-1/2 fermions on an 
anisotropic triangular lattice, i.e. with unequal hopping 
amplitudes, tf(b).i 7^ tf{b).2- The shape of the FS behaves 
as follows: For t/_2 > as one increases the chemical 
potential, the FS first breaks into four arcs at fif — 2tf i, 
and then breaks into six arcs at /i/ = 4t/.2 — 2iy^i, 
corresponding to the regimes IV- VI, in Fig. [T] b) and 
d). For tf^2 < tf.i the FS first breaks into two arcs 
at ^f — Atf,2 — 2t/.i, and then breaks into six arcs at 
/i/ = 2</_i coresponding to the regimes I-III, in Fig. [1] 
b) and c). At the special chemical potential ^/ = 2tf^i 
the FS is still nested, but there is only one nesting vector 
along the direction of the bonds with hopping amplitude 
tf^i. In the absence of the coupling to the BEC the phase 
diagram has a similar structure as for the isotropic case: 
s-wave pairing for attractive interaction, Fermi liquid be- 
havior for repulsive interaction, with the exception of the 
nested FS at = 2t/^i where one finds AF order (notice 
that in this case the filling is not 3/4). 

When the coupling to the bosons is turned on, one 
generates an even more complicated competition of pair- 
ing phases for repulsive Uf / in the vicinity of the point 
Hf — 2i/4, as is shown in Fig. U) Generally, for un- 
equal hopping the degeneracy between Di and D2 in 
P5p . as well as Px and Py is lifted: In the regime with 
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tf,2 > tfs < Di {D2) and (py) dominate. 

For tf 2 > tf,ii in the intermediate regime, in which the 
FS consists of four disjoined arcs, corresponding to the 
regime V in Fig. [Tl the type of ordering changes from 
Di to /. For tf^2 < the type of pairing also cventu- 
ahy becomes /-wave, but first develops two other types 
of pairing, in the regime II in Fig. [T] Firstly, one finds an 
unusual extended p-wave symmetry, which is schemati- 
cally shown in Fig. [2]f). Its wavefunction is of the form: 



s\n^e -7r/2 < 61 < 7r/2 
-s\T?e 7r/2 < 61 < 37r/2 



(16) 



The second type of pairing that appears before the 
system develops /-wave pairing is Di. These unusual 
pairing states are energetically favorable because of the 
anisotropic shape of the FS. For the regime in which the 
FS has just barely broken up into two arcs, the order pa- 
rameter assumes p-wave symmetry and the maxima are 
located along the y-axis, where the density of states is 
highest. As the region of open FS widens (see Fig. [5] 
f)), this pairing becomes energetically unfavorable, and 
the system develops Di-pairing, so that the maxima of 
the order parameter can again be located near the point 
of highest density of states. The energy gaps associated 
with these order parameters are of the same order of mag- 
nitude as for the isotropic lattice. 



VI. SPINLESS FERMIONS 

Finally, we consider a BFM with spinless fermions on 
an isotropic lattice. Due to the absence of s-wave scat- 
tering between fermions of the same spin state, there is 
no direct interaction, that is, Uff = Q. Hence, in the ab- 
sence of bosons, the spinless gas is non-interacting. The 
boson fluctuations, however, mediate an induced inter- 
action between the fermions. Due to the anti-symmetry 
of the Cooper pair wavefunction, pairing occurs in an 
odd angular momentum channel. We find a competition 
between p and f-wave pairing symmetry. For small to in- 
termediate fiUing {n < 0.65), p-wave pairing dominates. 
For larger fillings, for which the FS first approaches the 
shape of a hexagon and then breaks up into six arcs, the 
system shows /-wave pairing. Since these larger fillings of 
fermions are typically realized in the center of an atomic 
trap, this result would suggest a comparatively easy way 
to create an exotic pairing state experimentally. 

In contrast to this, a spinless BFM on a square lattice 
only shows p-wave pairing, since for the quadrangular 
shape of its FS, channels of higher angular momentum 
are of no advantage energetically. 



VII. CONCLUSION 

We have used a functional RG approach to study BFM 
of ultra-cold atoms, both with spin-1/2 and with spinless 



Px-BCS 




Pg^^-BCS Dl-BCS 



FIG. 4: Phase diagram of a Bose-Fermi mixture on an 
anisotropic triangular lattice. The vertical axis corresponds 
to the ratio t/,2/t/,i, the horizontal axis corresponds to the 
chemical potential Hf. The other parameters are given by 
V/tf = 3, Uff/tf^i = 2, and = 1 with a = 1,2). 



fermions, in a 2D optical lattice with triangular geom- 
etry. We found a number of competing types of order: 
For an isotropic lattice, we found pairing states with s, 
p-, extended d-, and /-wave symmetry, as well as AF 
order. For an anisotropic lattice, we additionally found 
extended p-wave pairing, and that the degeneracy be- 
tween the two types of extended d-wave pairing and the 
two types of p-wave pairing has been lifted. For a BFM 
with spinless fermions, we found a competition between 
p- and /-wave pairing. From the RG flow we also iden- 
tified the magnitude of the superconducting gaps, which 
turn out to be a fraction of the Fermi temperature, of 
the order of O.OITf-O.OOITf. Nevertheless, it is possible 
to increase these gaps substantially if the coupling to the 
BEG is in the strong-coupling regime. In this case, retar- 
dation effects can enhance the pairing mechanisms sub- 
stantially requiring an Eliashberg theory instead of BCS. 
To take into account retardation effects in this problem, 
the frequency dependence of U should also be kep^. 
The functional RG method with frequency dependence 
can be used in this regime. 

An important question in the context of ultracold 
atoms concerns the detectability of these phases. The 
exotic superconducting and AF states discussed in this 
work exhibit distinct experimental signatures. The AF 
state can be studied via time-of-flight images and Bragg 
scattering™, and the different superconducting phases 
can be measured through noise correlations^. Finally, 
laser stirring experiment*-^ can be used to establish the 
phase boundary between two phases. 

In conclusion, we presented a systematic study of the 
weak-coupling limit of BFMs of ultracold atoms in trian- 
gular optical lattices. For the various regimes, we found 
a number of exotic phases, and discussed their proper- 
ties and their relevance for ultra-cold atom experiments. 
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Given the generic structure of the Hamiltonian that we 
considered, the new types of phases, such as the extended 
d-wave pairing phase and the extended p-wave phases, are 
also of interest for the study of soHd-state systems with 



triangular symmetry. 
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